Papers with parametric models
Nonparametric Masked Language Modeling (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing language models (LMs) predict tokens with a softmax over a finite vocabulary, which can make it difficult to predict rare tokens or phrases. |
| Approach: | They introduce a nonparametric masked language model that replaces a softmax with a distribution over every phrase in a reference corpus and uses an in-batch approximation to train it. |
| Outcome: | The proposed model outperforms larger parametric models on 16 tasks including classification, fact probing and question answering. |
Non-Parametric Few-Shot Learning for Word Sense Disambiguation (2021.naacl-main)
Copied to clipboard
| Challenge: | Word sense disambiguation (WSD) is a problem in natural language processing . 84% of annotated words have less than 10 examples in the long-tail distribution . |
| Approach: | They propose a non-parametric few-shot learning approach to mitigate word sense disambiguation . they use a metric space to compute distances among the senses of a given word . |
| Outcome: | The proposed method achieves a 75.1 F1 score on the unified evaluation benchmark. |
Evaluation of Deep Gaussian Processes for Text Classification (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing models for text classification are limited by the expressability limit and require enormous empirical efforts to come up with a robust model architecture. |
| Approach: | They propose a Bayesian non-parametric Bayessian nonparametric model with strong function compositionality for the task of Text Classification. |
| Outcome: | The proposed models outperform shallow and deep Gaussian processes on the TREC (Text REtrieval Conference) datasets. |
An Efficient Memory-Augmented Transformer for Knowledge-Intensive NLP Tasks (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods rely on parametric models that store knowledge in parameters or retrieval-augmented models that have access to external knowledge sources. |
| Approach: | They propose a parametric parametric model that stores knowledge in its parameters or a retrieval-augmented model that has access to external knowledge sources. |
| Outcome: | The proposed method runs substantially faster across the board and produces more accurate results on WoW and ELI5. |
Privacy Implications of Retrieval-Based Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | a study of retrieval-based language models shows improved interpretability, factuality, and adaptability compared to parametric counterparts . kNN-LMs are more susceptible to leaking private information from their private datastore than parametric models . |
| Approach: | They present the first study of privacy risks in retrieval-based language models . they aim to strike a balance between utility and privacy in domains where privacy is of concern . |
| Outcome: | The proposed methods improve interpretability, factuality, and adaptability compared to parametric models . the study finds that kNN-LMs are more susceptible to leaking private data than parametric ones . |
Learn to Memorize: Scalable Continual Learning in Semiparametric Models with Mixture-of-Neighbors Induction Memory (2025.acl-long)
Copied to clipboard
| Challenge: | Semiparametric language models (LMs) use static storage, which lacks learning capability and is disconnected from the internal information flow of the parametric models. |
| Approach: | They reconceptualize the non-parametric memory represented by kNN-LM as a learnable Mixture-of-Neighbors Induction Memory (MoNIM) this synergizes the induction capabilities of attention heads with the memorization strength of feed-forward networks . |
| Outcome: | The proposed model is a learnable Mixture-of-neighbors induction memory (MoNIM) it synergizes the induction capabilities of attention heads with the memorization strength of feed-forward networks (FFNs). |